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2 FIELD OF THE INVENTION 



3 The present invention is directed toward the field of data retrieval. It is specifically directed 

4 toward graph algorithms to, automate organizing related search results. Primary computer and 

5 Web applications are in information, learning management, training, instruction, and education 

6 systems. 



7 BACKGROUND OF THE INVENTION 



r 

/ 



8 Data retrieval involves searching over a collection of objects to find a particular object or set of 

9 objects. The process of searching involves ranking objects in the collection with respect to a 

10 query to yield a list of relevant objects. Objects are digital data. Objects in a collection typically 

1 1 have a unique identifier. An ordering of relevant objects is computed to create a list of search 

12 results. The most relevant objects in the search results list are listed first. The ranking of objects 

13 on the search result list is based upon the relative match of the query to the digital data. Data 

14 retrieval may involve searching over a collection of objects in a database and returning a set of 

1 5 objects that match a query exactly based upon attributes and relationships between objects. 

1 6 Objects may be in a media format other than text, such as audio, video, or image. Time-var 5 dng 

1 7 media such as audio and video have a time duration. If the field is information retrieval, objects 

18 are typically documents stored in files containing information and the query takes the form of 

1 9 key words (“keywords”) or phrases entered by users. The query keywords are typically matched 

20 against the words in the document file. 

21 One common form of organization for objects in an ordered set. A set may be partially or totally 

22 ordered. Objects within a set may be sorted using a comparison fimction. A comparison 
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1 function may sort objects along a particular scale. Data retrieval systems may filter objects fi'om 

2 the search results into subsets of objects. Data retrieval systems may sort objects fi'om the search 

3 results list alphabetically by title, numerically by date, or by using any other ordering. 

4 The present invention applies to data retrieval. We require that objects are ranked, but the 

5 ranking could be based upon the object’s digital data or metadata associated with the object and 

6 contained in a metadata description. A metadata description is typically stored in a metadata file. 

7 Metadata is data that is typically associated a particular object in the collection, which could be 

8 entered manually by a human editor or computed automatically. The metadata file may include 

9 one or more categories. Categories may group objects by topic, discipline, concept, or other 

1 0 affinity, often according to an agreed upon taxonomy or classification system. Human editors or 

1 1 automated computer programs may classify objects into categories. The metadata file may be 

12 stored in a database. Metadata includes metadata elements (also called properties or attributes) 

13 and their values. Metadata may describe the content, quality, condition, type, format, duration, 

14 level of detail, level of complexity, level of interactivity, role, human language, geographical 

15 coverage, cultural aspects, version, level of interactivity, density, and use of data. Metadata 

16 values may be ordinal, numeric, or other data types. Metadata values may be selected fi'om a 

1 7 finite vocabulary. Metadata values may have a preferred or natural order. 

1 8 Data retrieval systems may organize objects by category and use categories to improve the 

19 display and access to objects. Objects may be classified into one or more categories. A category 

20 contains zero or more objects. Objects may be rank ordered within categories. Data retrieval 

2 1 systems may include categories related to one another in a graph and organize objects by the 

22 graph or use the graph to improve the display and access to objects. A graph is comprised of 

23 nodes (also called vertices) and edges. A graph is either directed or undirected. A path in a 

24 graph is a sequence of edges connecting nodes in a directed graph. A path data structure may 

25 include the nodes connected by edges comprising the path. A graph may be connected or 

26 unconnected. A graph of categories and relationships may be called a semantic network, topic 
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1 graph, or metathesaurus. Several algorithms exist to automatically classify documents into 

2 categories and to find objects by traversing a graph of categories and relationships. 

3 Computer systems storing a collection of objects often require human editors who manually 

4 assemble objects fi’om the collection into a set or sets of objects to group, filter, arrange, 

5 orgamze, or list the objects for human users. Human editors may also create new objects and 

6 create links fi-om object to another. On the World-Wide Web (Web) network of computer 

7 systems, for example, objects are Web resources. Each Web resource is identified by its Uniform 

8 Resource Identifier (URI). Web resources that have hyperlinks that are contained in Web 

9 resources, also called Web pages, to link to other related Web resources. Human users of the 

10 Web are typically also editors who manually create hyperlinks between related Web resources. 

1 1 Finding relevant objects and deciding how to assemble and link these objects to created sets of 

1 2 related objects is a time-consuming and manual process. 

1 3 Computer systems may orgamze documents, media, and other documents by accessing the 

1 4 object’s metadata description. Time- varying media, for example, may have a metadata 

1 5 description that includes a duration and computer systems displaying time-varying media may 

1 6 sorted by objects by this duration. Computer systems may store the ordered set in a structured 

1 7 data file. For example, computer systems may store an ordered set of links to other files in a file 

18 in a structured data file format, such as XML, providing data elements in a hierarchical structure. 

1 9 There are many computer applications requiring structured data files with links to other data 

20 files. Computer applications providing means for storing and accessing electronic books, for 

2 1 example, typically organize files containing parts of the book into a table of contents file having 

22 links to part of the book to provide for easy access to each part and forming a coherent set that 

23 has a preferred order for reading. 

24 Learning management, instruction, training, and education systems provide, deliver, or offer 

25 courses to users. These systems interact with a user or users to provide instruction or enable 
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1 learning using documents, media, and other digital objects on a computer or over the Internet to a 

2 user accessing a Web browser program on a computer. These systems may also be called 

3 Electronic learning (e-learmng) systems. Electronic learning systems often provide a course 

4 outline that consists of an orgamzed list or hierarchy of lessons. Electronic learning systems 

5 often allow the system or the user to initiate the process of navigating to a given lesson. Lessons 

6 are typically listed in a preferred learning order. Lessons are typically linked to the next lesson 

7 and previous lesson in the preferred learning order. Electronic learning systems typically store 

8 the ordered set of lessons in a structured data file format. 

9 The IMS Global Learning Consortium, Inc. (IMS), has developed the IMS Content Packaging 

10 specification for aggregating assembled learning materials into an interoperable, distributable 

1 1 package. The package includes an ordered set of objects in an organization structure that can be 

1 2 stored by an electronic learning system and offered, provided, or delivered to a user or users in 

13 the form of a course. The IMS Content Packaging information model specification provides 

14 data elements for multiple hierarchical organization structures and multiple item data elements 

1 5 within each organization that have identifiers that are references to Web resources. A default 

1 6 primary organization structure is identified. Each Web resource can be a object with its 

1 7 metadata stored according to a particular schema. 

18 Organizing objects for learning management, training, education, and instruction is usually 

19 performed by course developer accessing a document, media, or other application for creating 

20 and editing courses. The course developer takes into account the anticipated learning needs of 

21 the audience, their level of prior knowledge or skill, the amount of time available, and other 

22 factors. The course developer creates new courses by choosing appropriate media and designing 

23 appropriate lessons, activities, and other interactions for learners. The process of creating 

24 courses is time-consuming and can involve recreating lessons, activities, and other interactions 

25 and media resources that may already exist in some form elsewhere. 
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1 A collection of resources stored a database or file in the form of learning objects can allow the 

2 course developer using an editing application, database application, or other computer 

3 application to reuse resources fi*om prior courses, thus reducing the cost of developing new 

4 courses. Associating metadata with each resource in the collection can help course developers 

5 identify and find learning objects of the appropriate duration, difficulty, resource type, media 

6 type, level of detail and other attributes to server a role as a component to assemble a new 

7 course. Course developers may identify topic, task, or other types of categories for objects and 

8 identify introduction, motivation, or other roles for each object within each category. Categories 

9 and roles provide for placement of objects within the or ganisat ion structure of a course. For 

10 example, categories are formed into lessons organized as an ordered set within a course. 

1 1 Learning objects serving a particular role in a category can be formed into modules, units, or 

12 other parts within a lesson. Other organization structures are possible. 

13 The Institute for Electronic Engineers, Inc. (IEEE) has formed the Learning Technology 

14 Standards Committee (LTSC) to develop accredited technical standards for lea rnin g technology. 

15 The American National Standards Institute (ANSI) approved a standard data model for Learning 

16 Object Metadata (LOM, 1484.12.1) submitted by LTSC. The LOM standard provides data 

1 7 elements for Learning Resource Type ( resource type). Difficulty, Typical Learning Time 

1 8 (duration), techmcal format (media type), as well as title, description, keywords, and other 

1 9 metadata. An XML format for learmng object metadata files enables e-leaming systems to 

20 exchange learning object metadata between computer systems and applications conforming to the 

21 standard. Any object with a learning object metadata file is termed a “learning object”. Learning 

22 objects and their associated learning object metadata can be stored in a repository. A repository 

23 is a database or collection of files containing metadata. A repository references local or remote 

24 (Web) objects (resources) and metadata. A learning object content repository references both 

25 learning objects and learning object metadata. A learning object may be executed as program 

26 code in a Web browser providing a duration of execution and for interaction with the user. A 

27 learmng object may communicate with a learning management system using an agreed upon data 

28 commumcation protocol. Data communicated to a learning management system may be used by 
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1 the learning management system to select the next learning object a particular order within a 

2 course. 

3 Electronic learning systems may provide their users with access to a search feature for fin din g 

4 lessons, media, activities, or other learning objects. Electronic learning systems may allows 

5 users to specify a query to find learning objects of an appropriate duration, difficulty, media type, 

6 or other metadata stored in the metadata files associated with each learning object. However, 

7 learning objects are not typically automatically assembled, organized, and formed into a course 

8 in response to a user query. 

9 SUMMARY OF THE INVENTION 

10 Thus to overcome these problems, in a first aspect the present invention maps objects fi"om 

1 1 search results to one or more categories in a graph, computes statistics for each category based 

12 upon the mapped objects, and then uses both the statistics for each category and the relationships 

1 3 between categories encoded as the edges of the graph to find a best path through the graph. The 

14 best path is computed using Dijkastra’s graph search algorithm. The cost of traversing an edge 

15 in the graph is based upon the relevance of objects mapped to that category. Paths are evaluated 

16 using a measure of coherence. Coherent paths have the fewest number of breaks. A break in the 

1 7 path happens when a category does not have enough relevant objects. 

18 In a second aspect, the invention also provides a method to connect target objects. Target objects 

1 9 can be selected by a user, or can be selected fi'om the most relevant objects fi"om the search 

20 results. Target objects are mapped to specific categories that need to be connected to form a 

21 coherent path. The statistics for each category are updated to include the number and total 

22 relevant score of relevant objects. The graph traversal stops when all target objects have been 

23 connected. If the target objects are all within a depth limit, the best path is a minimum spanning 
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1 tree. However, if some targets are beyond the depth limit, then the algorithm starts traversing the 

2 graph again from another target until all target objects have been visited. 

3 In a further aspect, the invention also provides a method to sort a set of objects, obtaining an 

4 ordered set that is used to provide an improved progression through the related objects. The 

5 order of the objects is derived from the an ordering of the metadata vocabulary. The invention 

6 provides means of sorting objects according to a partial ordering of category metadata values, a 

7 partial ordering of role metadata values, and other partial orderings of object metadata 

8 vocabulary values. In an information system providing access to electronic books, the invention 

9 provides a means to create a table of contents with links to each section of the book In a 

1 0 electronic learning system, the invention provides means to create an organization structure to 

1 1 form a course. 

12 In an example embodiment, the present invention provides a method for data retrieval. The 

1 3 method comprising creating a set of related objects from a collection of objects. The creation 

14 includes the steps of: searching for a list of relevant objects and obtaining a rank-ordered list; 

1 5 selecting any target objects from the rank-ordered list, mapping the relevant objects in the 

16 rank-ordered list into categories and computing category statistics; connecting the categories into 

1 7 paths in a graph, [the graph having a node for each category and edges based upon category 

1 8 relationships]; choosing a best path in the graph based upon a path evaluation criterion; and 

19 selecting objects in categories on the best path based upon an object selection criterion. 

20 Also in the example embodiment, the present invention provides a system for assembling 

2 1 learning objects from a repository into an organization structure to form a course. The system 

22 provides a means for sorting the particular learning objects according to learning object metadata, 

23 creating an organizational structure for the particular learning objects, and displaying the 

24 organizational structure in the form of course. 
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1 BRIEF DESCRIPTION OF THK DRAWINGS 

2 These and other aspects, features, and advantages of the present invention will become apparent 

3 upon further consideration of the following detailed description of the invention when read in 

4 conjimction with the drawing figures, in which: 

5 Fig. 1 shows a flowchart of a method of creating a set of related objects; 

6 Fig. 2 shows a flowchart of a step of mapping a rank-ordered list into categories; 

7 Fig. 3 shows a flowchart of a step of connecting categories into a best path in a graph; 

8 Fig. 4 shows a flowchart of a step of evaluating a choosing a best path in the graph; 

9 Fig. 5 shows a flowchart of a step of selecting particular objects in categories on the best path; 

10 Fig. 6 shows a flowchart of a step of searching for objects and obtaining a rank-ordered list of 

1 1 related objects; 

12 Fig. 7 shows a flowchart of a method of assembling an ordered set of related objects from a 

1 3 collection of objects; 

14 Fig. 8 shows a flowchart of a step of sorting objects by comparing a position of metadata values; 

15 Fig. 9 shows a system for assembling learning objects to form a course; 

16 Fig. 10 shows a screen of a system for entering a query; 

17 Fig. 1 1 shows a screen of a system for selecting objects with a metadata file; 
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1 Fig. 12 shows a screen of a system for linking to objects in an ordered aggregation; 

2 Fig. 13 shows a data structure for storing a metadata; 

3 Fig. 14 shows a data structure for storing a rank-ordered list of relevant objects; 

4 Fig. 1 5 shows a data structure for storing a category-object mapping; 

5 Fig. 16 shows a data structure for storing category data; 

6 Fig. 17 shows a data structure for storing a graph of category nodes with relationship edges; 

7 Fig. 18 shows a data structure for storing paths; 

8 Fig. 1 9 shows a data structure for storing path data; 

9 Fig. 20 shows a data structure for storing an ordered aggregation of objects; 

10 Fig. 21 shows a flowchart of a step of selecting target objects; 

1 1 Fig. 22 shows a data structure for storing target objects; 

1 2 Fig. 23 shows a diagram of a repository storing a collection of objects and metadata files; 

13 Fig. 24 shows a data structure for storing a preferred order for metadata vocabulary; 

14 Fig. 25 shows a data structure for query parameters and values; 
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1 Fig. 26 shows a data structure for storing a list of object scores; 

2 Fig. 27 shows a flowchart of a method of selecting objects for an overview treatment of a best 

3 path; 

4 Fig 28 shows a flowchart of a method of selecting objects for an indepth treatment of a best path; 

5 and 

6 Fig. 29 shows a flowchart of a method of assembling objects into an ordered set in response to a 

7 query. 

8 DETAILED DESCRTPTTQN OF THF. INVENTION 

9 This invention provides methods, apparatus and systems for automatically or ganisin g search 

1 0 results. It provides methods to create a set of objects from a collection of objects using a data 

1 1 retrieval algorithm. It describes how to search for relevant objects from the collection to match a 

12 query, map search results to categories, traverse a graph connecting these categories to find a 

1 3 best path according to various heuristic criteria, and then select objects on the path to fit various 

14 parameters. 

1 5 For a collection of objects that have a duration, the invention provides a parameter can be 

1 6 adjusted to limit the total duration of set of objects on the path. If an overview treatment is 

1 7 needed, a parameter can be set to select the most relevant objects from each of several categories 

18 on the path. If an indepth treatment is needed, a method is provided to select objects in 

1 9 categories in close proximity to a focus category before proceeding to select objects from related 

20 categories on the path. 

2 1 This invention provides ways for selecting objects as targets for the graph traversal. The 

22 traversal algorithm assembles a set of objects on the path by attempting to reach any specified 
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target objects and by penalizing breaks in the path with nodes having categories with only a few 
relevant objects. If no target objects are selected, the target objects are set to the most relevant 
search results. 

This invention provides a process for mapping objects from the search results to one or more 
categories in a graph. The step of mapping includes computing statistics for each category based 
upon the objects contianed within categories. 

The invention provides ways to connect nodes into paths. In an example embodiment, the 
starting node is the category containing the most target objects and if there are two categories 
with an equivalent number of target objects, then the category with the highest total relevance 
score is chosen. The target nodes are the categories with a non-zero number of target objects. 

The best (lowest cost) path is computed using Dijkastra’s graph search algorithm. The 
algorithm maintains a queue and a visited list. At any node in the graph, there is a list of child 
nodes. The cost of traversing a link in the graph is given by a cost function. The child nodes are 
sorted by cost. If multiple nodes have an equivalent cost, all of the nodes are traversed. The 
algorithm traverses child nodes with the greatest number of target objects. If an equivalent 
number of target objects are mapped to multiple child nodes, the node with the greatest total 
relevance score is traversed first. If a target is reached, it is removed from the list of remaining 
targets and the graph search terminates. Any child category nodes with no relevant objects are 
traversed, but a break count is incremented. The graph traversal stops when all target objects 
have been reached or a depth limit has been met. If all the target objects are reached, the best 
path is a minimum spanning tree. However, if some targets are beyond the depth limit, then the 
algorithm traverses the graph starting at the next best start node as determined by a list of 
remaining targets. The process continues until the remaining targets have been visited. 

An evaluation process provides ways of choosing a best path from the paths list. Paths are 
evaluated using a measure of coherence. Coherent paths connect the most number of target 
objects. For paths with an equivalent number of target objects, coherent paths have the fewest 
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1 number of breaks. For paths with an equivalent number of breaks, the path with the highest sum 

2 of target relevance scores is chosen as the most coherent. Given the best path, the invention 

3 provides ways to select objects mapped to categories on a path based upon parameters provided 

4 in the query and on the statistics collected for each category by the mapping process. The 

5 invention provides an overview method to selects the most relevant objects from each category. 

6 The invention also provides an indepth method to chooses the most relevant objects from a given 

7 category before including objects from other categories. The choice of method is controlled by a 

8 search scope parameter that is part of the query. 

9 The invention also provides a method to sort objects according a preferred order for metadata 

1 0 vocabulary. In an example embodiment, the method sorts the selected objects by the earliest 

1 1 position of the object’s categories in an ordering of categories, then sorts within categories by the 

1 2 earliest position of the object’s role in a role ordering. The method provides ways of sorting 

1 3 objects according to any ordinal value with a given order, such as a scale. In alternate 

1 4 embodiments, a difficulty scale or resource type order could be used. 

1 5 This invention also describes a system for assembling learning objects from a repository into an 

1 6 organization structure in response to a query and forming a course to display. The system 

1 7 provides a user interface screen for entering query keywords and query parameters to search for 

1 8 learning objects. The query interface screen provides ways of limiting the search for learning 

19 objects to particular types of learning resources, those fulfilling only particular instructional roles 

20 within the created course, and those within a specified range of difficulty. A process is described 

2 1 for creating an organizational structure for the learning objects [ including placing learning 

22 objects into this structure ]. The query user interface screen provides ways of adjusting the 

23 duration of the created course based upon the estimated learning time of learning objects placed 

24 in the organization structure as determined by duration metadata stored in the learning object 

25 metadata file for each learning object. 

26 The organization structure is suitable for loading into a learning management system. The 

27 organization structure is used to create a dynamic Web page as a course outline and set of 
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1 additional Web pages that link learning objects in a particular order to form a course pages in a 

2 format accessible over the Web from a Web browser. In an alternate embodiment, objects are 

3 executable programs that run in a Web browser and commxmicate with a management system. 

4 For example, objects could be learning objects that execute a program designed to track learner 

5 progress during an activity. Learning objects communicate the user’s performance data during 

6 the activity to the learning management system. The learning management system uses the 

7 performance data to determine the next learning object to execute. 

8 This invention provides advantages for information and data retrieval systems that display search 

9 results that are related to one another: 

10 1 ) Information and data retrieval algorithms usually rank objects based on relevance to 

1 1 the query. Relevance ranking may result in a set of independent, unrelated, or even 

1 2 incoherent search results. Users of information and data retrieval systems may want to 

1 3 view particular related objects order orders a coherent group in the list of search results. 

14 This invention solves this problem by using both metadata associated with objects and 

1 5 relationships between the metadata to assemble related objects to improve coherence. 

16 2) Information and data retrieval algorithms group search results by category or list the 

1 7 search results by rank. It is often difficult for users to get an overview of the relevant 

1 8 objects because the most relevant objects appear in many different categories and are thus 

19 not proximal to one another. This invention solves this problem by creating groups of 

20 objects based upon their relevance ranking, associated metadata including categories, and 

2 1 the proximity of these categories in a graph of category relationships. 

22 

23 

24 

25 



3)Information and data retrieval systems usually retrieve search results based upon a 
query. The search results may include objects from many categories. However, it is often 
difficult for users to specify whether few relevant objects from each of many related 
categories (overview) or many relevant objects from each of few related categories 
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(indq)th) are preferred. The present invention solves this problem by providing a search 
scope as part of the query that determines which of two methods for selecting objects is 

3 used. 

4 4) Information and data retrieval systems can require users to specify categories as part of 

5 the query to focus the search for related objects. However, it is often difficult for users to 

6 select particular categories without first viewing particular objects and their respective 

7 categories. The present invention solves this problem by providing a method of 

8 using user selections of target particular objects to identify a set of categories as start 

9 nodes, then searching the graph of category nodes and category relationship edges in 

1 0 vicinity proximity of the start nodes using a greedy graph search algorithm to find a 

1 1 minimum spanning tree connecting these nodes in a best path. 

1 2 This invention also provides several advantages for electronic learning systems that provide 

1 3 learning obj ects to a user: 

14 1) The duration of an course is usually determined at the time the course, or any other 

1 5 desired organization structure ordered, is constructed. However, users may want to size 

1 6 the course to their available time. This invention provides an algorithm that can attempt 

1 7 to size a course comprised of learning objects to a desired mi nim u m time, maximum 

1 8 time, or time duration based upon the duration of selected learning objects. 

19 

20 
21 
22 
23 



2) The organization of learning objects within an course, or any other desired structure 
ordered, is usually performed manually. This manual organization process is 
time-consuming and non-uniform. This invention provides an algorithm that can sort 
selected learning objects quickly and uniformly into a progression and place them in a 
desired structure without human intervention. 
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1 This method is efficient with respect to current practice. One reason for this efficiency is 

2 because in current practice, metadata is derived automatically for each object or entered manually 

3 in parallel, by independent human editors, while object creation and assembly cannot be done 

4 independently and in parallel across objects because these processes involve the step of manually 

5 linking objects. In the present invention, a new method is possible whereby the links needed for 

6 assembly, sorting, and linking are isolated in the graph of categories and category relationships 

7 instead of the objects. In the new practice, the schema for the metadata in the repository is 

8 designed, the possible vocabulary for use in metadata files is developed, and then the 

9 relationships between the metadata vocabulary are designed for the entire repository, even before 

10 objects are identified. The practice is efficient because values fi"om the vocabulary for the 

1 1 metadata are often repeated across metadata files because common metadata entry and editing 

1 2 tools are used by human editors, thus the effort to update the relationships between the metadata 

1 3 vocabulary is less than the effort involved in linking objects. 

14 A example embodiment is shown in Figure 1 . A search step (100) creates a rank-ordered list of 

1 5 relevant objects fi'om a collection (120). A target selection step (300) chooses a subset of target 

16 objects fi’om the rank-ordered list. A mapping step (200) computes statistics for categories of 

17 objects in the rank-ordered list, including any target objects. A connecting step (400) traverses a 

1 8 graph (41 0) to generate a set of paths cormecting the categories, including an evaluation step to 

19 choose a best path (500). An object selection step (600) selects particular objects on the best 

20 path to obtain the desired set of related search results (620). 

21 An example embodiment of the search process (100) is shown in Figure 6. A prior art scoring 

22 process (170) computes a relevance score in the Object scrollable (140) for each object in the 

23 repository based upon a match to the query (140). The example embodiment combines any text 

24 fiom the Object file (150) and an XML form of the Metadata file into a combined XML file, 

25 produces a fast lookup index fiom the metadata values and text terms in the combined XML file, 

26 and then looks up the values of query parameters in the index. In the example embodiment, the 

27 query parameters include keywords (111), desired resource types and media type (1 14), and 
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1 desired difficulty levels (1 16). Alternate embodiments match only the metadata values, only the 

2 ‘ text, and features of the object extracted and stored as metadata values. In the example 

3 embodiment, the query keyword parameters are matched using a weighted match based upon the 

4 frequency of text terms in the combined XML file and other query parameter values must match 

5 exactly with metadata values in the combined XML file. In other embodiments, a weighted 

6 score balancing the different query parameters may be used. A ranking process (180) lists the 

7 object identifiers (141) for the relevant objects in order by score in the rank-ordered list (144) 

8 An example embodiment is shown in Figure 2. The mapping process (200) takes as input the 

9 Target Objects List (310) from the target selection process and the rank-ordered list (144) from 

10 the search process. For each object identifier (151) listed in the rank-ordered list, the mapping 

1 1 process looks up the metadata file (130) for each object identifier (151) in the target objects list, 

1 2 looks up the category metadata values (1 3 1 ) in the given metadata file, and then stores the 

1 3 unique obj ect identifier (151) with respect to each category (22 1 ) in the Category-Obj ect Table 

14 (220). In an alternate embodiment, the classification into categories is computed from the digital 

1 5 data of the object. 

1 6 The Compute Category Statistics process (225) updates the Category Data Table (230). For each 

1 7 category, it computes the number of objects mapped to the category (232), the sum of the 

1 8 relevance scores of all objects mapped to the category (233), and the average relevance score 

1 9 (234). If there are target selections, the mapping process also computes the number of target 

20 objects per category (235). 

21 An example embodiment is shown in Figure 3. The connecting process (400) is a graph traversal 

22 algorithm that operates on a Graph (410) resulting in a Paths list (441) connecting nodes in the 

23 graph starting at a start node (415). The start node is the category with the highest number of 

24 target objects (235). If there are no target objects, the start node is a focus category having the 

25 highest sum of scores (233). Next, the connecting process selects the highest remaining targets 

26 (485) after removing the start node (475). At each iteration, the remaining targets (485) with less 
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1 than a minimum, C, number of objects (232) are put to the bottom of the list so that they are not 

2 preferred as start nodes. This constant controls how much evidence, in the form of mapped 

3 objects, is required to determine the focus of the graph search. In the example embodiment, C is 

4 set to the constant 3. The graph search (486) searches to connect the remaining targets (485) 

5 starting at the start node. The graph search traverses the graph using a prior art greedy graph 

6 search algorithm that maintains a queue of nodes and a visited list. Nodes on the visited list are 

7 not traversed again 

8 Each Path (430) in the Paths list (441) includes category nodes (431) and category relationship 

9 edges (432). The Graph is represented in a table (410) where a one (1) means an directed edge 

10 between the category nodes on the row to the category node on the column, a zero (0) means no 

1 1 edge. In the example embodiment, the Graph is a connected, directed graph G = (V, E) where 

12 the vertices, V, are the set of category nodes (41 1) and the edges, E, are the set of relationships 

1 3 between categories. The example embodiment uses the Resource Description Framework (RDF) 

14 format to store the graph in a file and then loads the RDF file to create a set of connected graph 

1 5 node and edges. The nodes of the graph are categories fi-om the a fixed category vocabulary. 

1 6 There is an edges in the graph between two category nodes if there is some kind of category 

1 7 relationship between the two nodes. A category relationship between two categories is a 

1 8 semantic notion of relatedness, such as one category being part of , a subclass of another 

19 category. 

20 The traversal algorithm generates a Paths list (441) connecting categories through relationship 

21 edges. In the example embodiment, each time a node is added to the Paths list, the path length 

22 (442), the total of the category relevance scores (445), total duration (446), and number of objects 

23 (447) is updated in a Path Data Table (440). If a node added to the path is a category that has less 

24 than a constant M number of relevant objects mapped (232), the number of breaks (443) is 

25 increased. The minimum threshold M in the example embodiment is set to a small constant that 

26 provides the minimum number of objects per category required for minimal coherence, one (1). 

27 Other embodiments may compute a minimum number by assuming that the objects are randomly 
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1 selected from a known distribution of objects per categories and identifying a break as a 

2 deviation from the expected distribution. 

3 The graph search algorithm terminates at a fixed depth in the graph, a constant D. In the 

4 example embodiment, D is set to a small constant reflecting the desired number of categories, 

5 four (4). If target objects were selected, the graph search algorithm may terminate prior to the 

6 fixed depth D if all target objects were connected on the given path. If the fixed depth was 

7 reached and not all target objects were selected, each of the categories (131) of the target object 

8 with the highest rank (142) on the Search results table (140) are pushed on the queue and the 

9 traversal algorithm is restarted with the starting node set to the highest scoring of these categories 

10 in the Category Data Table (230), the depth reinitialized to zero (0). The algorithm terminates 

1 1 when there are no more targets (450). 

12 An example embodiment is shown in Figure 4. The choosing process (500) applies an evaluation 

13 criteria to choose the best path (510) among the paths returned by the connecting process (441). 

14 Because of the depth limit, the graph traversal algorithm cannot be guaranteed to return the least 

15 cost (best) path. Therefore, a set of heuristic evaluation criteria are used to return a best path. 

16 First, if target objects were selected, that paths that have the most number of target objects are 

1 7 selected. If two best paths have an equivalent highest number of target objects or there are no 

18 target objects, the paths with the fewest number of breaks (443) are selected. If two best paths 

19 are still otherwise equivalent, the paths with the highest total score (445) are selected. 

20 Objects are selected (600) using an object selection criteria. In the example embodiment, if there 

21 are target objects the Target objects (311) are added to the Selected Objects Set (610). If the user 

22 specified an overview search scope, the method iterates over the Best Path Nodes (5 1 2) selecting 

23 objects from the Category-Object Table (220) with the highest relevance rank (142) in the Search 

24 results table (140), without the sum of the total duration (446) of said selected objects exceeding 

25 the maximum duration (112). If the user specified an indepth seareh scope, the method selects 

26 the objects in the Category-Object Table (220) listed with category (221) equal to the start node 
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1 (510) in rank (142) order without the sum of the total duration (446) of said selected objects 

2 exceeding the maximimi duration (112). In another embodiment, a bin packing algorithm could 

3 be used to maximize the relevance of the objects fitting within the desired duration. 

4 This invention also provides an assembly method (800). The assembly method provides a step 

5 for sorting (700) the output of the selection process into an Ordered Set of Objects (730). An 

6 example embodiment of the assembly process is shown in Figure 29. An example embodiment 

7 of the assembly process without the optional step of selecting target objects (900) is shown in 

8 Figure 8. An example embodiment of the sorting step is shown in Figure 7. An example 

9 embodiment of the user flow through the screens of the assembly process without the optional 

1 0 step of selecting target objects (910) is shown in Figure 9. In the example embodiment shown in 

1 1 Figure 29, the sorting executes a total ordering of the selected objects set (610). The selected 

1 2 objects in the set are first sorted by their relative positions in a category order (7 1 0). In the 

1 3 example embodiment, the category order is total order of traversal of the category nodes in the 

1 4 graph. The order of the categories may reflect pedagogical principles. Each object has 

1 5 associated metadata that includes a role (136), also called an “intended use” in Figure 1 0. The 

16 selected objects are further sorted within each said category by the relative position of said 

1 7 object’s role (136) in a specified role order (720). In the example embodiment, the role order is a 

1 8 total order. The role order may reflect rhetorical, cognitive, or domain-specific principles. In the 

1 9 example embodiment, our rhetorical principles put “introduction” first and “conclusion” last. 

20 The cognitive principles put the roles “motivation”, “concepts”, and “procedures” in the 

21 specified order. The domain-specific principles put “scenarios”, “definitions”, “architecture”, 

22 “system”, and “code listing” in the specified order. These principles may be combined to 

23 determine said total order. In other embodiments, the role order may be determined relative to . 

24 the particular categories in the graph. In the example embodiment, the Ordered Set of Objects 

25 (730) is stored in an XML file encoded in the IMS Content Packaging format. Each Web 

26 resource is listed in the resource element (733) 1 .6.2 in the Href attribute (1 .6.2.3). The order of 

27 each item element (732) 1 .5.2.4 is given in that item’s position within an organization element 

28 (73 1 ) 1 .5.2. The aggregation is displayed as a Web page of hyperlinks to each objects and the 
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1 hyperlinks are listed top to bottom. Figure 12 shows the Web browser user interface with the 

2 ordered aggregation as a course outline Web page. Each Web resource becomes a hyper linked 

3 lesson and the lessons are displayed order specified by the sorting process. In other 

4 embodiments, the ordered aggregation may be a course comprised of learning objects that are 

5 launched by a learmng management system, where said learning management system decides on 

6 the next learning object based upon the order of the learning objects in the course. 

7 An example embodiment is shown in Figure 21 . In the example embodiment, the target selection 

8 process selects targets fi-om the rank-ordered list (1441) in the Search Result Table (140) as 

9 target objects for the connection process (400). This process is executed only if the user selects 

1 0 the “manual assembly” option (1 1 8) when specifying a query (Figure 1 0). The output of the 

1 1 search process (100) is then a target objects list (310) of learning objects (1 59) that is displayed 

12 to the user (Figure 11). In the example embodiment, selected values fi'om each object’s metadata 

1 3 description file ( 1 30) are displayed (Figure 1 1 ), such as its categories (131), duration (132), 

14 media (133), resource types (134), difficulty level (135), and roles (136). The user then selects 

1 5 one or more target object selections (158) fi'om the list of relevant objects. ( 1 59). The unique 

16 object identifiers of the target objects (159) are stored in the Target Objects List (310). 

1 7 Variations described for the present invention can be realized in any combination desirable for 

1 8 each particular application. Thus particular limitations, and/or embodiment enhancements 

1 9 described herein, which may have particular advantages to the particular application need not be 

20 used for all applications. Also, not all limitations need be implemented in methods, systems 

2 1 and/or apparatus including one or more concepts of the present invention. 

22 The object (150) may be a document, media, Web resource, or other digital entity. For web 

23 resources, the unique object identifier (151) is specified the URI (uniform resource identifier) 

24 format. The object data (152) may be stored as digital information in a variety of formats 

25 (HTML, MPEG, JPEG, etc.) that can be stored, retrieved, and displayed fi'om a file or files stored 

26 on a computer. If the object is a Web resource, the object data can be stored on a local or a 
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1 remote server. The object data can be retrieved by its URI. In the example embodiment, the 

2 object data is stored in a repository (120) of files on a local server computer. 

3 In the example embodiment, the rank-ordered list or related objects (144) is the result of a user 

4 query (110) using a prior art search algorithm. The rank-ordered list is a total order of unique 

5 identifiers ( 1 5 1 ) to objects that is ordered by rank and stored in the Search Results Table ( 1 40). 

6 The objects are related to the query. A query is a set of parameters and parameter values. One of 

7 the keyword parameters is keywords and its values are keywords (1 1 1). The objects contain 

8 terms (513) such as words, phrases, and other strings. A score (153) is computed for each object 

9 that indicates the relevance of the object’s terms (513) to the keywords (1 1 1) in the query. In one 

10 embodiment, an index maps from the terms (153) to unique identifiers (151) and the search 

1 1 process use the index to look up objects given a query. Figure 10 shows a Web browser user 

1 2 interface displaying the user keywords ( 1 1 1 ) of typed into a query field. 

13 Categories (131) in the Metadata description (130) also are identified as as Category Nodes (21 1) 

14 in the Graph (410) and are identified as strings in the Category order (710). In other 

1 5 embodiments, the categories may also be the values of a category query parameter. The 

1 6 categories may be strings that convey topics (e.g., “websphere”, “j2ee”), tasks (e.g., “install”, 

17 “administer”, “uninstall”), or other distinctions. The categories should be relatively semantically 

1 8 separable, so that it is easy to tell whether a given object belongs in a category. 

19 Roles (136) in the Metadata description (130) are also identified as strings in the Roles Order 

20 (720). The example embodiment roles are strings from the following list of strings: 

21 (“introduction”, “motivation”, “scenarios”, “concepts”, “definitions”, “architecture”, “system”, 

22 “procedure”, “code listing”, “conclusion”). 

23 A metadata description-file (1 30) is digital information about an object that is separate from the 

24 object data (152). Metadata may be stored in a variety of formats. In the example embodiment, 

25 the metadata description-file for each object is a document stored in a file in the repository (120) 
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1 in the IEEE Learning Object Metadata (LOM) 6.4 XML format. The XML data in the file is 

2 encoded in elements that have tags with optional attributes that delimit the metadata values. The 

3 metadata values are accessed by parsing the XML file. The categories (131) metadata values are 

4 one or more values taken fi'om a vocabulary of categories and are stored in the element 

5 [Classification|TaxonPath|Taxon|ld, 9.2.2. 1]. The roles (136) are one or more values taken fi'om 

6 a vocabulary of categories and are stored in the element [Educational|Leaming Resource Type 

7 5.2]. The duration is stored in [Educational|Typical Learning Time, 5.9]. A useful media 

8 vocabulary is taken fi'om MIME Part Two IETF RFC 246 (http://www.ietf.org/rfc/rfc2046.txt ) 

9 and is stored in [Technical|Format, 4.1]. A one-to-one mapping between object identifiers (151) 

1 0 and metadata description-files (130) is needed. In the example embodiment, the mapping is 

1 1 stored in the Obj ect-Metadata Mapping Table (160). 

12 In an alternate embodiment, the collection of objects is stored in a database. The database holds 

1 3 the object data (152) and the metadata description-file (130) for each object (150). A database 

14 query language is used to access metadata values in the metadata description-file (130). 

15 In the example embodiment, the user query includes additional preferences as user criteria. The 

1 6 search process weighs the additional preferences to update the relevance score (153) that is stored 

17 in the Search Results Table (140). There are additional preferences for desired media (1 14), 

1 8 resource types, (115), and difficulty levels (116). A prior art search algorithm matches the 

19 desired media and resource types (1 14), desired roles (115) and difficulty levels (116) against the 

20 media (133), resource types (134), roles (136), difficult levels (135), or other metadata stored in 

2 1 the metadata description-file. Figure 1 0 shows a Web browser user interface showing a query 

22 including selections of parameters for desired media and resource types (114), desired roles (136) 

23 labeled as “intended use” , desired difficulty levels (116) and other parameters. 

24 In the example embodiment, the user query includes a set of user constraints on the linked search 

25 results. There is a duration range (1 12) menu with a minimum and maximum duration and a 

26 search scope setting (113) menu with possible values of “overview” and “indepth”. The duration 
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1 range menu offers the user duration ranges from 1 minute in length to two hours in length on an 

2 increasing scale, each duration being roughly twice as long as the previous but rounded to 

3 convenient values. Other embodiments offer a preference for the number of categories (the path 

4 length), level of detail, or other user criteria. Figure 10 shows a Web browser user interface with 

5 the duration range (112) entered by a user in a field labeled : Desired Course Duration” , and a 

6 search scope (1 13) in a field labeled “Desired Search Scope” displaying the “overview” setting. 

7 The present invention can be realized in hardware, software, or a combination of hardware and 

8 software. A visualization tool according to the present invention can be realized in a centralized 

9 fashion in one computer system, or in a distributed fashion where different elements are spread 

1 0 across several interconnected computer systems. Any kind of computer system - or other 

1 1 apparatus adapted for carrying out the methods and/or ftmctions described herein - is suitable. A 

12 typical combination of hardware and software could be a general purpose computer system with 

13 a computer program that, when being loaded and executed, controls the computer system such 

1 4 that it carries out the methods described herein. The present invention can also be embedded in a 

1 5 computer program product, which comprises all the features enabling the implementation of the 

1 6 methods described herein, and which - when loaded in a computer system - is able to carry out 

1 7 these methods. 

1 8 Computer program means or computer program in the present context include any expression, in 

1 9 any language, code or notation, of a set of instructions intended to cause a system having an 

20 information processing capability to perform a particular fimction either directly or after 

21 conversion to another language, code or notation, and/or reproduction in a different material 

22 form. 

23 Thus the invention includes an article of manufacture which comprises a computer usable 

24 medium having computer readable program code means embodied therein for causing a fimction 

25 described above. The computer readable program code means in the article of manufacture 

26 comprises computer readable program code means for causing a computer to effect the steps of a 
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1 method of this invention. Similarly, the present invention may be implemented as a computer 

2 program product comprising a computer usable medium having computer readable program code 

3 means embodied therein for causing a a flmction described above. The computer readable 

4 program code nieans in the computer program product comprising computer readable program 

5 code means for causing a computer to effect one or more functions of this invention. 

6 Furthermore, the present invention may be implemented as a program storage device readable by 

7 machine, tangibly embodying a program of instructions executable by the machine to perform 

8 method steps for causing one or more functions of this invention. 

9 It is noted that the foregoing has outlined some of the more pertinent objects and embodiments of 

1 0 the present invention. This invention may be used for many applications. Thus, although the 

1 1 description is made for particular arrangements and methods, the intent and concept of the 

12 invention is suitable and applicable to other arrangements and applications. It will be clear to 

1 3 those skilled in the art that modifications to the disclosed embodiments can be effected without 

14 departing from the spirit and scope of the invention. The described embodiments ought to be 

1 5 construed to be merely illustrative of some of the more prominent features and applications of the 

1 6 invention. Other beneficial results can be realized by applying the disclosed invention in a 

1 7 different marmer or modifying the invention in ways known to those familiar with the art. 



DOCKET NUMBER: YOR920040099US1 



-24- 




